{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###                                             Task Background and Data Pre-processing\n",
    "#### this jupyter notebook is used to pre-processing data. at the end you will have vocabulary, labels, training/validation/test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " in this task, you will be asked to predict top 5 topics given a question and its description.  \n",
    " \n",
    "###### source files: \n",
    " \n",
    " 1. question_train_set.txt.  you can get question id and its string information, you can transform it to train_X.\n",
    " \n",
    "     it has 5 columns, each is split with '\\t'. format as below:\n",
    "     \n",
    "     question_id ct1,ct2,ct3,...,ctn wt1,wt2,wt3,...,wtn cd1,cd2,cd3,...cdn wd1,wd2,wd3,...,wdn\n",
    "     \n",
    "     second column is character token of title, third column is word token of title, forth column is character token of description, fifth column is \n",
    "     \n",
    "     word token of description.\n",
    "     \n",
    " 2. question_topic_train_set.txt.  you can get question id and its labels. you can transform it to  train_Y. \n",
    " \n",
    "     topics associated with a question. it contains with two columns, each column is splitted with '\\t'. \n",
    "     \n",
    " 3. question_eval_set.txt.  you can get question id and its string information, this will be valid_X. this is same format as question_train_set.txt\n",
    " \n",
    " \n",
    "###### additional stats information:\n",
    "\n",
    "1. averaged_length:\n",
    "\n",
    "   {'desc_char': 117.39879138670524, 'title_char': 22.207077611187056, 'desc_word': 58.272774333851004, 'title_word': 12.841507923253822}\n",
    "\n",
    "2. averaged length of a input. total length of all information(words,character of title+desc): 210.\n",
    "\n",
    "3. word of title+desc: 71\n",
    "\n",
    "4. character of title+desc: 139\n",
    "\n",
    "5. as can see from word embeding files, there are about 11k of charactor tokens, and 410k of word tokens that frequency is more than 5 times\n",
    "\n",
    "  in the data set.\n",
    "  \n",
    "6. total unique labels: 1999\n",
    "\n",
    "###### basic processes\n",
    "\n",
    "1. in this notebook, we will use character token of title and description. max sequence length will be set to 200. any sequence exceed of it, will be\n",
    "\n",
    "   truncated, any sequence short of it, will be padded. \n",
    "\n",
    "2. we will generate vocabulary/ labels dict, and training/validation/test data, then save to cache file(as a pickle file), so during training we can\n",
    "\n",
    "   load it quickly.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "import package successful...\n"
     ]
    }
   ],
   "source": [
    "# import some packages\n",
    "import pandas as pd\n",
    "from collections import Counter\n",
    "from tflearn.data_utils import pad_sequences\n",
    "import random\n",
    "import numpy as np\n",
    "import h5py\n",
    "import pickle\n",
    "print(\"import package successful...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('train_data_x:', (2999967, 5))\n",
      "('train_data_y:', (2999967, 2))\n",
      "('valid_data_x:', (217360, 5))\n"
     ]
    }
   ],
   "source": [
    "# read source file as csv\n",
    "base_path='data/ieee_zhihu_cup/'\n",
    "train_data_x=pd.read_csv(base_path+'question_train_set3.txt',sep='\\t', encoding=\"utf-8\")\n",
    "train_data_y=pd.read_csv(base_path+'question_topic_train_set3.txt',sep='\\t', encoding=\"utf-8\")\n",
    "valid_data_x=pd.read_csv(base_path+'question_eval_set3.txt', sep='\\t',encoding=\"utf-8\")\n",
    "\n",
    "train_data_x=train_data_x.fillna('')\n",
    "train_data_y=train_data_y.fillna('')\n",
    "valid_data_x=valid_data_x.fillna('')\n",
    "print(\"train_data_x:\",train_data_x.shape)\n",
    "print(\"train_data_y:\",train_data_y.shape)\n",
    "print(\"valid_data_x:\",valid_data_x.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question_id</th>\n",
       "      <th>title_char</th>\n",
       "      <th>title_word</th>\n",
       "      <th>desc_char</th>\n",
       "      <th>desc_word</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6555699376639805223</td>\n",
       "      <td>c324,c39,c40,c155,c180,c180,c181,c17,c4,c1153,...</td>\n",
       "      <td>w305,w13549,w22752,w11,w7225,w2565,w1106,w16,w...</td>\n",
       "      <td>c335,c101,c611,c189,c97,c144,c147,c101,c15,c76...</td>\n",
       "      <td>w231,w54,w1681,w54,w11506,w5714,w7,w54,w744,w1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2887834264226772863</td>\n",
       "      <td>c44,c110,c101,c286,c106,c150,c101,c892,c632,c1...</td>\n",
       "      <td>w377,w54,w285,w57,w349,w54,w108215,w6,w47986,w...</td>\n",
       "      <td>c1265,c518,c74,c131,c274,c57,c768,c769,c368,c3...</td>\n",
       "      <td>w12508,w1380,w72,w27045,w276,w111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-2687466858632038806</td>\n",
       "      <td>c15,c768,c769,c1363,c650,c1218,c2361,c11,c90,c...</td>\n",
       "      <td>w875,w15450,w42394,w15863,w6,w95421,w25,w803,w...</td>\n",
       "      <td>c693,c100,c279,c99,c189,c532,c101,c189,c145,c1...</td>\n",
       "      <td>w140340,w54,w48398,w54,w140341,w54,w12856,w54,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-5698296155734268</td>\n",
       "      <td>c473,c1528,c528,c428,c295,c15,c101,c188,c146,c...</td>\n",
       "      <td>w8646,w2744,w1462,w9,w54,w138,w54,w50,w110,w14...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-6719100304248915192</td>\n",
       "      <td>c190,c147,c105,c219,c220,c101,c647,c219,c220,c...</td>\n",
       "      <td>w380,w54,w674,w133,w54,w134,w614,w54,w929,w307...</td>\n",
       "      <td>c644,c1212,c253,c199,c431,c452,c424,c207,c2,c1...</td>\n",
       "      <td>w4821,w1301,w16003,w928,w1961,w2565,w50803,w11...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           question_id                                         title_char  \\\n",
       "0  6555699376639805223  c324,c39,c40,c155,c180,c180,c181,c17,c4,c1153,...   \n",
       "1  2887834264226772863  c44,c110,c101,c286,c106,c150,c101,c892,c632,c1...   \n",
       "2 -2687466858632038806  c15,c768,c769,c1363,c650,c1218,c2361,c11,c90,c...   \n",
       "3    -5698296155734268  c473,c1528,c528,c428,c295,c15,c101,c188,c146,c...   \n",
       "4 -6719100304248915192  c190,c147,c105,c219,c220,c101,c647,c219,c220,c...   \n",
       "\n",
       "                                          title_word  \\\n",
       "0  w305,w13549,w22752,w11,w7225,w2565,w1106,w16,w...   \n",
       "1  w377,w54,w285,w57,w349,w54,w108215,w6,w47986,w...   \n",
       "2  w875,w15450,w42394,w15863,w6,w95421,w25,w803,w...   \n",
       "3  w8646,w2744,w1462,w9,w54,w138,w54,w50,w110,w14...   \n",
       "4  w380,w54,w674,w133,w54,w134,w614,w54,w929,w307...   \n",
       "\n",
       "                                           desc_char  \\\n",
       "0  c335,c101,c611,c189,c97,c144,c147,c101,c15,c76...   \n",
       "1  c1265,c518,c74,c131,c274,c57,c768,c769,c368,c3...   \n",
       "2  c693,c100,c279,c99,c189,c532,c101,c189,c145,c1...   \n",
       "3                                                      \n",
       "4  c644,c1212,c253,c199,c431,c452,c424,c207,c2,c1...   \n",
       "\n",
       "                                           desc_word  \n",
       "0  w231,w54,w1681,w54,w11506,w5714,w7,w54,w744,w1...  \n",
       "1                  w12508,w1380,w72,w27045,w276,w111  \n",
       "2  w140340,w54,w48398,w54,w140341,w54,w12856,w54,...  \n",
       "3                                                     \n",
       "4  w4821,w1301,w16003,w928,w1961,w2565,w50803,w11...  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# understand your data: that's take a look of data\n",
    "train_data_x.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('dict_length_columns:', {'desc_char': 117.39879138670524, 'title_char': 22.207077611187056, 'desc_word': 58.272774333851004, 'title_word': 12.841507923253822})\n"
     ]
    }
   ],
   "source": [
    "# compute average length of title_char, title_word, desc_char, desc_word\n",
    "\n",
    "dict_length_columns={'title_char':0,'title_word':0,'desc_char':0,'desc_word':0}\n",
    "num_examples=len(train_data_x)\n",
    "train_data_x_small=train_data_x.sample(frac=0.01)\n",
    "for index, row in train_data_x_small.iterrows():\n",
    "    title_char_length=len(row['title_char'].split(\",\"))\n",
    "    title_word_length=len(row['title_word'].split(\",\"))\n",
    "    desc_char_length=len(row['desc_char'].split(\",\"))\n",
    "    desc_word_length=len(row['desc_word'].split(\",\"))\n",
    "    dict_length_columns['title_char']=dict_length_columns['title_char']+title_char_length\n",
    "    dict_length_columns['title_word']=dict_length_columns['title_word']+title_word_length\n",
    "    dict_length_columns['desc_char']=dict_length_columns['desc_char']+desc_char_length\n",
    "    dict_length_columns['desc_word']=dict_length_columns['desc_word']+desc_word_length\n",
    "dict_length_columns={k:float(v)/float(num_examples*0.01) for k,v in dict_length_columns.items()}\n",
    "print(\"dict_length_columns:\",dict_length_columns)\n",
    "\n",
    "# averaged length of a input. total length of all information(words,character of title+desc): 210.\n",
    "# word of title+desc: 71\n",
    "# character of title+desc: 139"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question_id</th>\n",
       "      <th>topic_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6555699376639805223</td>\n",
       "      <td>7739004195693774975,3738968195649774859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2887834264226772863</td>\n",
       "      <td>-3149765934180654494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-2687466858632038806</td>\n",
       "      <td>-760432988437306018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-5698296155734268</td>\n",
       "      <td>-6758942141122113907,3195914392210930723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-6719100304248915192</td>\n",
       "      <td>3804601920633030746,4797226510592237555,435133...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           question_id                                          topic_ids\n",
       "0  6555699376639805223            7739004195693774975,3738968195649774859\n",
       "1  2887834264226772863                               -3149765934180654494\n",
       "2 -2687466858632038806                                -760432988437306018\n",
       "3    -5698296155734268           -6758942141122113907,3195914392210930723\n",
       "4 -6719100304248915192  3804601920633030746,4797226510592237555,435133..."
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data_y.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('average_num_labels:', 2.3440333333333334)\n"
     ]
    }
   ],
   "source": [
    "# average labels for a input\n",
    "train_data_y_small=train_data_y.sample(frac=0.01)\n",
    "num_examples=len(train_data_y_small)\n",
    "num_labels=0\n",
    "for index, row in train_data_y_small.iterrows():\n",
    "    topic_ids=row['topic_ids']\n",
    "    topic_id_list=topic_ids.split(\",\")\n",
    "    num_labels+=len(topic_id_list)\n",
    "average_num_labels=float(num_labels)/float(num_examples)\n",
    "print(\"average_num_labels:\",average_num_labels)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question_id</th>\n",
       "      <th>title_char</th>\n",
       "      <th>title_word</th>\n",
       "      <th>desc_char</th>\n",
       "      <th>desc_word</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6215603645409872328</td>\n",
       "      <td>c924,c531,c102,c284,c188,c104,c98,c107,c11,c11...</td>\n",
       "      <td>w1340,w1341,w55,w1344,w58,w6,w24178,w26959,w47...</td>\n",
       "      <td>c1128,c529,c636,c572,c1321,c139,c540,c223,c510...</td>\n",
       "      <td>w4094,w1618,w20104,w19234,w1097,w1005,w4228,w2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6649324930261961840</td>\n",
       "      <td>c346,c1549,c413,c294,c675,c504,c183,c74,c541,c...</td>\n",
       "      <td>w40132,w1357,w1556,w1380,w2464,w33,w16791,w109...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-4251899610700378615</td>\n",
       "      <td>c96,c97,c97,c98,c99,c100,c101,c141,c42,c42,c10...</td>\n",
       "      <td>w53,w54,w1779,w54,w1309,w54,w369,w949,w65587,w...</td>\n",
       "      <td>c149,c148,c148,c42,c185,c95,c95,c186,c186,c186...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6213817087034420233</td>\n",
       "      <td>c504,c157,c221,c221,c633,c468,c469,c1637,c1072...</td>\n",
       "      <td>w5083,w12537,w10427,w29724,w6,w2566,w11,w18476...</td>\n",
       "      <td>c15,c131,c39,c40,c85,c166,c969,c2456,c17,c636,...</td>\n",
       "      <td>w2550,w24,w239,w98,w19456,w11,w108710,w3483,w2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-8930652370334418373</td>\n",
       "      <td>c0,c310,c35,c122,c123,c11,c317,c91,c175,c476,c...</td>\n",
       "      <td>w33792,w21,w83,w6,w21542,w21,w140670,w25,w1110...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           question_id                                         title_char  \\\n",
       "0  6215603645409872328  c924,c531,c102,c284,c188,c104,c98,c107,c11,c11...   \n",
       "1  6649324930261961840  c346,c1549,c413,c294,c675,c504,c183,c74,c541,c...   \n",
       "2 -4251899610700378615  c96,c97,c97,c98,c99,c100,c101,c141,c42,c42,c10...   \n",
       "3  6213817087034420233  c504,c157,c221,c221,c633,c468,c469,c1637,c1072...   \n",
       "4 -8930652370334418373  c0,c310,c35,c122,c123,c11,c317,c91,c175,c476,c...   \n",
       "\n",
       "                                          title_word  \\\n",
       "0  w1340,w1341,w55,w1344,w58,w6,w24178,w26959,w47...   \n",
       "1  w40132,w1357,w1556,w1380,w2464,w33,w16791,w109...   \n",
       "2  w53,w54,w1779,w54,w1309,w54,w369,w949,w65587,w...   \n",
       "3  w5083,w12537,w10427,w29724,w6,w2566,w11,w18476...   \n",
       "4  w33792,w21,w83,w6,w21542,w21,w140670,w25,w1110...   \n",
       "\n",
       "                                           desc_char  \\\n",
       "0  c1128,c529,c636,c572,c1321,c139,c540,c223,c510...   \n",
       "1                                                      \n",
       "2  c149,c148,c148,c42,c185,c95,c95,c186,c186,c186...   \n",
       "3  c15,c131,c39,c40,c85,c166,c969,c2456,c17,c636,...   \n",
       "4                                                      \n",
       "\n",
       "                                           desc_word  \n",
       "0  w4094,w1618,w20104,w19234,w1097,w1005,w4228,w2...  \n",
       "1                                                     \n",
       "2                                                     \n",
       "3  w2550,w24,w239,w98,w19456,w11,w108710,w3483,w2...  \n",
       "4                                                     "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_data_x.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question_id</th>\n",
       "      <th>topic_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6555699376639805223</td>\n",
       "      <td>7739004195693774975,3738968195649774859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2887834264226772863</td>\n",
       "      <td>-3149765934180654494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-2687466858632038806</td>\n",
       "      <td>-760432988437306018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-5698296155734268</td>\n",
       "      <td>-6758942141122113907,3195914392210930723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-6719100304248915192</td>\n",
       "      <td>3804601920633030746,4797226510592237555,435133...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           question_id                                          topic_ids\n",
       "0  6555699376639805223            7739004195693774975,3738968195649774859\n",
       "1  2887834264226772863                               -3149765934180654494\n",
       "2 -2687466858632038806                                -760432988437306018\n",
       "3    -5698296155734268           -6758942141122113907,3195914392210930723\n",
       "4 -6719100304248915192  3804601920633030746,4797226510592237555,435133..."
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data_y.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question_id</th>\n",
       "      <th>title_char</th>\n",
       "      <th>title_word</th>\n",
       "      <th>desc_char</th>\n",
       "      <th>desc_word</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6215603645409872328</td>\n",
       "      <td>c924,c531,c102,c284,c188,c104,c98,c107,c11,c11...</td>\n",
       "      <td>w1340,w1341,w55,w1344,w58,w6,w24178,w26959,w47...</td>\n",
       "      <td>c1128,c529,c636,c572,c1321,c139,c540,c223,c510...</td>\n",
       "      <td>w4094,w1618,w20104,w19234,w1097,w1005,w4228,w2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6649324930261961840</td>\n",
       "      <td>c346,c1549,c413,c294,c675,c504,c183,c74,c541,c...</td>\n",
       "      <td>w40132,w1357,w1556,w1380,w2464,w33,w16791,w109...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-4251899610700378615</td>\n",
       "      <td>c96,c97,c97,c98,c99,c100,c101,c141,c42,c42,c10...</td>\n",
       "      <td>w53,w54,w1779,w54,w1309,w54,w369,w949,w65587,w...</td>\n",
       "      <td>c149,c148,c148,c42,c185,c95,c95,c186,c186,c186...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6213817087034420233</td>\n",
       "      <td>c504,c157,c221,c221,c633,c468,c469,c1637,c1072...</td>\n",
       "      <td>w5083,w12537,w10427,w29724,w6,w2566,w11,w18476...</td>\n",
       "      <td>c15,c131,c39,c40,c85,c166,c969,c2456,c17,c636,...</td>\n",
       "      <td>w2550,w24,w239,w98,w19456,w11,w108710,w3483,w2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-8930652370334418373</td>\n",
       "      <td>c0,c310,c35,c122,c123,c11,c317,c91,c175,c476,c...</td>\n",
       "      <td>w33792,w21,w83,w6,w21542,w21,w140670,w25,w1110...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           question_id                                         title_char  \\\n",
       "0  6215603645409872328  c924,c531,c102,c284,c188,c104,c98,c107,c11,c11...   \n",
       "1  6649324930261961840  c346,c1549,c413,c294,c675,c504,c183,c74,c541,c...   \n",
       "2 -4251899610700378615  c96,c97,c97,c98,c99,c100,c101,c141,c42,c42,c10...   \n",
       "3  6213817087034420233  c504,c157,c221,c221,c633,c468,c469,c1637,c1072...   \n",
       "4 -8930652370334418373  c0,c310,c35,c122,c123,c11,c317,c91,c175,c476,c...   \n",
       "\n",
       "                                          title_word  \\\n",
       "0  w1340,w1341,w55,w1344,w58,w6,w24178,w26959,w47...   \n",
       "1  w40132,w1357,w1556,w1380,w2464,w33,w16791,w109...   \n",
       "2  w53,w54,w1779,w54,w1309,w54,w369,w949,w65587,w...   \n",
       "3  w5083,w12537,w10427,w29724,w6,w2566,w11,w18476...   \n",
       "4  w33792,w21,w83,w6,w21542,w21,w140670,w25,w1110...   \n",
       "\n",
       "                                           desc_char  \\\n",
       "0  c1128,c529,c636,c572,c1321,c139,c540,c223,c510...   \n",
       "1                                                      \n",
       "2  c149,c148,c148,c42,c185,c95,c95,c186,c186,c186...   \n",
       "3  c15,c131,c39,c40,c85,c166,c969,c2456,c17,c636,...   \n",
       "4                                                      \n",
       "\n",
       "                                           desc_word  \n",
       "0  w4094,w1618,w20104,w19234,w1097,w1005,w4228,w2...  \n",
       "1                                                     \n",
       "2                                                     \n",
       "3  w2550,w24,w239,w98,w19456,w11,w108710,w3483,w2...  \n",
       "4                                                     "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_data_x.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0, 'PAD')\n",
      "(1, 'UNK')\n",
      "(2, 'CLS')\n",
      "(3, 'SEP')\n",
      "(4, 'unused1')\n",
      "(5, 'unused2')\n",
      "(6, 'unused3')\n",
      "(7, 'unused4')\n",
      "(8, 'unused5')\n",
      "(9, '</s>')\n",
      "vocabulary of char generated....\n"
     ]
    }
   ],
   "source": [
    " # create vocabulary_dict, label_dict, generate training/validation data, and save to some place \n",
    "    \n",
    " # create vocabulary of charactor token by read word_embedding.txt \n",
    "word_embedding_object=open(base_path+'unused_current/char_embedding.txt')\n",
    "lines_wv=word_embedding_object.readlines()\n",
    "word_embedding_object.close()\n",
    "char_list=[]\n",
    "char_list.extend(['PAD','UNK','CLS','SEP','unused1','unused2','unused3','unused4','unused5'])\n",
    "PAD_ID=0\n",
    "UNK_ID=1\n",
    "for i, line in enumerate(lines_wv):\n",
    "    if i==0: continue\n",
    "    char_embedding_list=line.split(\" \")\n",
    "    char_token=char_embedding_list[0]\n",
    "    char_list.append(char_token)    \n",
    "    \n",
    "# write to vocab.txt under data/ieee_zhihu_cup\n",
    "vocab_path=base_path+'vocab.txt'\n",
    "vocab_char_object=open(vocab_path,'w')\n",
    "\n",
    "word2index={}\n",
    "for i, char in enumerate(char_list):\n",
    "    if i<10:print(i,char)\n",
    "    word2index[char]=i\n",
    "    vocab_char_object.write(char+\"\\n\")\n",
    "vocab_char_object.close()\n",
    "print(\"vocabulary of char generated....\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(u'7476760589625268543', 2308)\n",
      "(u'4697014490911193675', 1746)\n",
      "(u'-4653836020042332281', 1579)\n",
      "(u'-8175048003539471998', 1475)\n",
      "(u'-8377411942628634656', 1382)\n",
      "(u'-7046289575185911002', 1338)\n",
      "(u'-5932391056759866388', 1283)\n",
      "(u'2787171473654490487', 1145)\n",
      "(u'-7129272008741138808', 1085)\n",
      "(u'2587540952280802350', 1079)\n",
      "(u'-4931965624608608932', 1079)\n",
      "(u'-6748914495015758455', 1049)\n",
      "(u'-5513826101327857645', 993)\n",
      "(u'2347973810368732059', 970)\n",
      "(u'9069451131871918127', 958)\n",
      "(u'-8132909213241034354', 904)\n",
      "(u'-3517637179126242000', 867)\n",
      "(u'-5872443091340192918', 834)\n",
      "(u'-3522198575349379632', 830)\n",
      "(u'1127459907694805235', 829)\n",
      "generate label dict successful...\n"
     ]
    }
   ],
   "source": [
    " # generate labels list, and save to file system \n",
    "c_labels=Counter()\n",
    "train_data_y_small=train_data_y[0:100000]#.sample(frac=0.1)\n",
    "for index, row in train_data_y_small.iterrows():\n",
    "    topic_ids=row['topic_ids']\n",
    "    topic_list=topic_ids.split(',')\n",
    "    c_labels.update(topic_list)\n",
    "\n",
    "label_list=c_labels.most_common()\n",
    "label2index={}\n",
    "label_target_object=open(base_path+'label_set.txt','w')\n",
    "for i, label_freq in enumerate(label_list):\n",
    "    label,freq=label_freq\n",
    "    label2index[label]=i\n",
    "    label_target_object.write(label+\"\\n\")\n",
    "    if i<20: print(label,freq)\n",
    "label_target_object.close()\n",
    "print(\"generate label dict successful...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('label_list_sparse:', array([1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.,\n",
      "       0., 0., 0.]))\n"
     ]
    }
   ],
   "source": [
    "def transform_multilabel_as_multihot(label_list,label_size):\n",
    "    \"\"\"\n",
    "    convert to multi-hot style\n",
    "    :param label_list: e.g.[0,1,4], here 4 means in the 4th position it is true value(as indicate by'1')\n",
    "    :param label_size: e.g.199\n",
    "    :return:e.g.[1,1,0,1,0,0,........]\n",
    "    \"\"\"\n",
    "    result=np.zeros(label_size)\n",
    "    #set those location as 1, all else place as 0.\n",
    "    result[label_list] = 1\n",
    "    return result\n",
    "\n",
    "label_list=[0,1,2,10]\n",
    "label_size=20\n",
    "label_list_sparse=transform_multilabel_as_multihot(label_list,label_size)\n",
    "print(\"label_list_sparse:\",label_list_sparse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def get_X_Y(train_data_x,train_data_y,label_size, test_mode=False):\n",
    "    \"\"\"\n",
    "    get X and Y given input and labels\n",
    "    input:\n",
    "    train_data_x:\n",
    "    train_data_y:\n",
    "    label_size: number of total unique labels(e.g. 1999 in this task)\n",
    "    output:\n",
    "    X,Y\n",
    "    \"\"\"\n",
    "    X=[]\n",
    "    Y=[]\n",
    "    if test_mode:\n",
    "        train_data_x_tiny_test=train_data_x[0:1000] # todo todo todo todo todo todo todo todo todo todo todo todo \n",
    "        train_data_y_tiny_test=train_data_y[0:1000] # todo todo todo todo todo todo todo todo todo todo todo todo \n",
    "    else:\n",
    "        train_data_x_tiny_test=train_data_x\n",
    "        train_data_y_tiny_test=train_data_y\n",
    "\n",
    "    for index, row in train_data_x_tiny_test.iterrows():\n",
    "        if index==0: continue\n",
    "        # get character of title and dssc\n",
    "        title_char=row['title_char']\n",
    "        desc_char=row['desc_char']\n",
    "        # split into list\n",
    "        title_char_list=title_char.split(',')\n",
    "        desc_char_list=desc_char.split(\",\")\n",
    "        # transform to indices\n",
    "        title_char_id_list=[vocabulary_word2index.get(x,UNK_ID) for x in title_char_list if x.strip()]\n",
    "        desc_char_id_list=[vocabulary_word2index.get(x,UNK_ID) for x in desc_char_list if x.strip()]\n",
    "        # merge title and desc: in the middle is special token 'SEP'\n",
    "        title_char_id_list.append(vocabulary_word2index['SEP'])\n",
    "        title_char_id_list.extend(desc_char_id_list)\n",
    "        X.append(title_char_id_list)\n",
    "        if index<3: print(index,title_char_id_list)\n",
    "        if index%100000==0: print(index,title_char_id_list)\n",
    "\n",
    "    for index, row in train_data_y_tiny_test.iterrows():\n",
    "        if index==0: continue\n",
    "        topic_ids=row['topic_ids']\n",
    "        topic_id_list=topic_ids.split(\",\")\n",
    "        label_list_dense=[label2index[l] for l in topic_id_list if l.strip()]\n",
    "        label_list_sparse=transform_multilabel_as_multihot(label_list_dense,label_size)\n",
    "        Y.append(label_list_sparse)\n",
    "        if index%100000==0: print(index,\";label_list_dense:\",label_list_dense)\n",
    "\n",
    "    return X,Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabulary_word2index['SEP']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_data(cache_file_h5py,cache_file_pickle,word2index,label2index,train_X,train_Y,vaild_X,valid_Y,test_X,test_Y):\n",
    "    # train/valid/test data using h5py\n",
    "    f = h5py.File(cache_file_h5py, 'w')\n",
    "    f['train_X'] = train_X\n",
    "    f['train_Y'] = train_Y\n",
    "    f['vaild_X'] = vaild_X\n",
    "    f['valid_Y'] = valid_Y\n",
    "    f['test_X'] = test_X\n",
    "    f['test_Y'] = test_Y\n",
    "    f.close()\n",
    "    # save word2index, label2index\n",
    "    with open(cache_file_pickle, 'ab') as target_file:\n",
    "        pickle.dump((word2index,label2index), target_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, [110, 143, 11, 31, 35, 28, 11, 522, 1392, 197, 667, 12, 194, 915, 1611, 509, 58, 67, 33, 15, 60, 64, 84, 1417, 648, 268, 66, 143, 109, 16, 3, 543, 96, 64, 26, 73, 19, 67, 33, 363, 601, 16])\n",
      "(2, [58, 67, 33, 2152, 562, 1354, 822, 12, 137, 1690, 165, 13, 134, 95, 93, 12, 356, 529, 43, 119, 16, 3, 624, 24, 91, 120, 106, 203, 11, 106, 52, 106, 14, 120, 120, 359, 11, 55, 24, 14, 401, 52, 11, 14, 21, 11, 37, 11, 90, 57, 83, 21, 36, 52, 11, 14, 83, 34, 11, 52, 21, 57, 55, 52, 11, 76, 359, 11, 20, 28, 11, 3662, 11, 20, 27, 11, 90, 57, 83, 21, 36, 52, 11, 345, 742, 84, 669, 239, 36, 21, 21, 55, 185, 38, 38, 39, 39, 39, 35, 79, 14, 52, 21, 46, 57, 401, 30, 34, 52, 35, 57, 46, 79, 38, 42, 57, 83, 21, 24, 83, 21, 38, 28, 28, 38, 24, 83, 38, 624, 24, 91, 120, 106, 203, 431, 28, 23, 906, 431, 28, 23, 310, 426, 624, 455, 38, 624, 24, 91, 120, 106, 203, 431, 28, 23, 906, 431, 28, 23, 310, 426, 624, 455, 431, 28, 23, 30, 83, 431, 28, 23, 318, 83, 91, 14, 83, 21, 52, 35, 36, 21, 90, 120])\n",
      "(3, [260, 715, 587, 235, 99, 58, 11, 30, 331, 36, 57, 83, 24, 11, 289, 647, 131, 145, 223, 1140, 342, 299, 16, 1218, 209, 1480, 326, 647, 241, 430, 12, 191, 155, 137, 136, 16, 3])\n",
      "(4, [90, 14, 42, 402, 357, 11, 438, 402, 357, 11, 440, 435, 11, 294, 460, 47, 19, 131, 145, 575, 124, 396, 564, 12, 265, 64, 651, 299, 16, 3, 495, 394, 351, 865, 93, 175, 60, 300, 18, 145, 468, 921, 10, 600, 67, 33, 64, 12, 11, 111, 216, 1628, 239, 86, 14, 11, 36, 46, 24, 91, 50, 25, 36, 21, 21, 55, 185, 38, 38, 39, 39, 39, 35, 251, 36, 30, 36, 106, 35, 42, 57, 90, 38, 244, 106, 24, 52, 21, 30, 57, 83, 38, 28, 28, 20, 23, 49, 37, 37, 37, 25, 11, 42, 120, 14, 52, 52, 50, 25, 30, 83, 21, 24, 46, 83, 14, 120, 25, 85, 411, 30, 83, 34, 57, 39, 11, 294, 460, 19, 131, 145, 575, 124, 396, 564, 12, 265, 64, 651, 299, 16, 86, 38, 14, 85])\n",
      "(100000, [131, 43, 715, 1137, 12, 227, 95, 147, 533, 1672, 16, 3, 155, 41, 273, 760, 1243, 1137, 126, 10, 349, 230, 160, 274, 267, 531, 121, 715, 1137, 10, 207, 74, 97, 26, 232, 18, 118, 131, 43, 715, 1137, 1585, 227, 147, 92, 16])\n",
      "(200000, [18, 109, 243, 199, 308, 96, 138, 263, 229, 351, 398, 134, 351, 16, 3, 18, 123, 243, 12, 199, 308, 11, 96, 138, 28, 62, 285, 263, 229, 351, 398, 191, 188, 309, 19, 351, 12, 199, 308, 16])\n",
      "(300000, [716, 565, 573, 607, 453, 375, 839, 19, 67, 33, 400, 1273, 136, 16, 3, 103, 141, 369, 504, 573, 607, 10, 1377, 95, 375, 839, 96, 209, 19, 573, 607, 71, 533, 15, 92, 99, 60, 453, 134, 84, 597, 11, 168, 13, 19, 145, 573, 607, 114, 11, 1172, 843, 573, 10, 19, 145, 11, 667, 573, 11, 573, 607, 792, 277, 11, 114, 573, 104, 268, 694, 375, 839, 659, 851, 15, 18, 102, 10, 11, 19, 67, 33, 400, 1273, 89, 609, 136, 16, 11, 67, 33, 80, 181, 114, 1172, 843, 573, 16, 573, 607, 94, 102, 792, 277, 11, 114, 268, 694, 12, 573, 16, 11, 56, 92, 336, 99, 60, 716, 565, 573, 607, 453, 134, 84, 11, 201, 336, 12, 11, 15, 659, 985, 12, 11, 192, 94, 33, 245, 16])\n",
      "(400000, [32, 390, 457, 252, 12, 289, 159, 175, 449, 115, 361, 110, 19, 131, 145, 265, 410, 12, 121, 177, 16, 3])\n",
      "(500000, [258, 121, 32, 135, 1095, 238, 118, 12, 41, 264, 343, 198, 376, 436, 89, 74, 505, 53, 194, 156, 517, 64, 136, 16, 3])\n",
      "(600000, [96, 138, 547, 372, 18, 43, 1294, 66, 142, 12, 65, 938, 16, 3])\n",
      "(700000, [58, 67, 33, 354, 1494, 158, 292, 113, 56, 342, 173, 593, 84, 41, 99, 13, 878, 212, 235, 1113, 301, 764, 16, 3, 15, 917, 13, 132, 363, 10, 252, 228, 310, 113, 175, 12, 71, 13, 10, 63, 337, 13, 315, 51, 61, 113, 99, 174, 265, 89, 74, 117, 53, 593, 84, 41, 99, 13, 878, 212, 235, 1113, 301, 764, 12, 623, 599, 22, 11, 268, 66, 262, 617, 170, 743, 1187, 743, 12, 479, 320, 92, 26, 10, 54, 113, 63, 327, 43, 92, 182, 252, 856, 356, 18, 800, 12, 479, 158, 775, 144, 11, 86, 30, 90, 79, 11, 52, 46, 42, 50, 25, 14, 20, 91, 23, 37, 40, 31, 44, 76, 20, 14, 40, 34, 37, 34, 40, 14, 34, 31, 24, 37, 91, 44, 76, 44, 20, 27, 14, 20, 20, 28, 23, 25, 11, 34, 14, 21, 14, 45, 46, 14, 39, 39, 30, 34, 21, 36, 50, 25, 40, 28, 29, 25, 11, 34, 14, 21, 14, 45, 46, 14, 39, 36, 24, 30, 79, 36, 21, 50, 25, 49, 29, 44, 25, 85])\n",
      "(800000, [40, 62, 12, 589, 298, 10, 192, 15, 192, 173, 142, 16, 3])\n",
      "(900000, [590, 605, 19, 92, 201, 278, 1101, 550, 1964, 142, 16, 110, 334, 1104, 314, 1211, 92, 201, 16, 467, 110, 334, 1104, 260, 143, 12, 1964, 142, 110, 19, 1101, 550, 1964, 142, 33, 16, 3])\n",
      "(1000000, [476, 1232, 423, 341, 159, 38, 317, 1131, 96, 138, 32, 635, 84, 339, 1012, 16, 3, 86, 36, 20, 85, 11, 17, 97, 453, 476, 1232, 423, 341, 125, 13, 137, 336, 403, 157, 87, 1569, 87, 1569, 187, 187, 187, 86, 38, 36, 20, 85, 11, 17, 109, 41, 99, 501, 116, 61, 54, 189, 129, 59, 159, 1131, 10, 63, 168, 13, 172, 221, 41, 743, 743, 1631, 144, 10, 71, 59, 41, 579, 56, 317, 258, 154, 423, 341, 1631, 144, 22, 22, 22, 22, 22, 11, 214, 395, 100, 32, 156, 151, 33, 201, 151, 33, 999, 10, 328, 15, 89, 60, 19, 41, 18, 112, 476, 1631, 144, 22, 22, 22, 22, 11, 105, 61, 54, 78, 453, 54, 629, 469, 64, 87, 56, 80, 123, 293, 71, 87, 731, 264, 413, 453, 10, 134, 84, 53, 282, 59, 159, 1131, 71, 59, 317, 1131, 22, 22, 22, 22, 11, 66, 108, 99, 476, 674, 17, 100, 149, 171, 453, 318, 215, 271, 73, 341, 333, 22, 22, 22, 22, 22, 89, 13, 54, 272, 102, 17, 249, 133, 59, 646, 813, 10, 73, 261, 17, 328, 56, 731, 264, 22, 22, 22, 22])\n",
      "(1100000, [56, 284, 427, 129, 11, 270, 1516, 11, 238, 962, 389, 16, 3])\n",
      "(1200000, [58, 138, 126, 658, 1665, 144, 56, 92, 15, 64, 1592, 507, 16, 3])\n",
      "(1300000, [47, 26, 56, 75, 10, 163, 140, 59, 19, 274, 267, 12, 41, 68, 3, 17, 348, 174, 464, 15, 557, 274, 267, 18, 43, 41, 13, 67, 33, 163, 140, 26, 10, 168, 464, 124, 621, 110, 12, 80, 181, 274, 267, 115, 17, 108, 535, 113, 18, 43, 126, 95, 10, 151, 71, 13, 263, 621, 343, 107, 248, 12, 10, 621, 18, 356, 18, 545, 410, 53, 146, 140, 124, 146, 377, 344, 10, 53, 71, 19, 171, 15, 274, 267, 146, 10, 140, 124, 19, 171, 137, 877, 10, 15, 127, 148, 263, 67, 33, 80, 181, 63, 107, 248, 274, 267, 146, 26, 10, 151, 80, 181, 1713, 15, 124, 80, 80, 948, 948, 104, 146, 32, 18, 220, 10, 270, 184, 54, 13, 18, 471, 1181, 589, 22, 11, 137, 110, 12, 80, 181, 19, 171, 103, 1539, 1301, 790, 10, 17, 71, 15, 127, 148, 17, 137, 110, 12, 80, 181, 53, 487, 19, 59, 19, 274, 267, 115, 616, 10, 19, 115, 151, 33, 286, 43, 126, 95, 104, 17, 223, 283, 65, 10, 17, 63, 116, 146, 108, 65, 10, 370, 95, 26, 65, 163, 10, 19, 80, 17, 140, 124, 17, 15, 13, 274, 267, 146, 108, 12, 10, 125, 13, 282, 97, 104, 146, 108, 32, 18, 220, 10, 100, 32, 56, 75, 26, 10, 17, 231, 97, 10, 56, 701, 270, 80, 17, 320, 1901, 51, 1301, 790, 10, 168, 97, 19, 43, 41, 1203, 1196, 606, 1576, 10, 56, 701, 15, 13, 164, 12, 274, 267, 425, 22, 11, 100, 32, 17, 32, 18, 207, 281, 354, 330, 56, 75, 227, 126, 188, 260, 188, 1066, 10, 17, 108, 535, 227, 126, 188, 18, 188, 529, 10, 19, 87, 92, 210, 75, 293, 124, 377, 344, 12, 10, 17, 71, 19, 115, 286, 246, 18, 410, 886, 186, 12, 151, 183, 150, 170, 10, 125, 13, 200, 93, 282, 59, 26, 54, 183, 163, 140, 10, 22, 17, 164, 12, 15, 127, 148, 10, 17, 97, 660, 18, 471, 589, 298, 10, 125, 13, 164, 12, 348, 174, 59, 19, 26, 621, 110, 151, 183, 150, 170, 12, 163, 140, 26, 22, 11, 17, 192, 94, 33, 245, 273])\n",
      "(1400000, [137, 260, 238, 95, 10, 116, 196, 110, 493, 56, 75, 130, 325, 428, 56, 75, 130, 929, 861, 56, 75, 130, 690, 110, 354, 400, 56, 75, 130, 339, 205, 349, 304, 56, 75, 130, 1467, 162, 56, 75, 192, 96, 138, 2869, 563, 16, 3, 155, 41, 58, 508, 313, 1241, 137, 238, 12, 511, 339, 157, 354, 126, 95, 10, 931, 702, 351, 49, 29, 23, 711, 717, 10, 89, 74, 161, 393, 12, 284, 129, 1836, 472, 56, 701, 19, 341, 337, 204, 330, 130, 382, 75, 330, 130, 421, 784, 330, 130, 761, 785, 330, 22, 241, 192, 73, 453, 450, 576, 707, 10, 495, 394, 147, 65, 60, 638, 134, 143, 22, 11, 635, 47, 78, 1598, 572, 998, 753, 10, 1374, 184, 188, 309, 825, 1124, 10, 15, 127, 511, 56, 127, 160, 13, 408, 60, 263, 156, 1078, 381, 204, 672, 130, 374, 640, 75, 640, 130, 75, 389, 636, 235, 1768, 884, 130, 75, 389, 1002, 299, 130, 284, 129, 149, 264, 296, 152, 175, 116, 74, 47, 286, 207, 56, 75, 110, 12, 18, 207, 271, 286, 207, 153, 134, 173, 802, 16, 15, 1365, 163, 901, 22, 11, 746, 10, 116, 196, 157, 354, 126, 95, 12, 284, 129, 327, 563, 10, 56, 109, 19, 67, 33, 476, 674, 136, 16])\n",
      "(1500000, [96, 138, 322, 60, 32, 417, 530, 88, 53, 126, 230, 160, 16, 3, 348, 174, 28, 44, 26, 10, 490, 129, 361, 927, 2619, 10, 881, 430, 105, 189, 10, 1061, 762, 186, 430, 645, 10, 371, 140, 124, 103, 141, 410, 115, 12, 126, 95, 201, 10, 277, 150, 166, 12, 328, 201, 10, 17, 71, 127, 148, 59, 19, 249, 197, 229, 3729, 10, 125, 13, 105, 13, 163, 140, 126, 95, 19, 271, 92, 271, 201, 987, 515, 103, 141, 487, 480, 12, 880, 171, 10, 17, 192, 94, 33, 245])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1600000, [135, 1054, 1612, 53, 12, 325, 339, 10, 430, 109, 15, 114, 737, 10, 924, 527, 524, 81, 61, 168, 60, 892, 536, 10, 54, 337, 15, 337, 1703, 1383, 16, 3, 17, 32, 135, 1054, 47, 1612, 986, 53, 18, 43, 18, 667, 12, 2029, 144, 10, 270, 392, 109, 78, 236, 3782, 1897, 657, 854, 10, 63, 13, 78, 17, 406, 118, 375, 213, 746, 194, 18, 299, 430, 292, 10, 322, 104, 1612, 986, 53, 12, 430, 292, 18, 220, 172, 17, 1691, 115, 84, 22, 54, 13, 408, 277, 195, 10, 17, 192, 96, 138, 937, 698, 16, 868, 612, 61, 18, 118, 10, 924, 527, 12, 241, 116, 204, 421, 164, 12, 15, 189, 10, 1612, 986, 53, 12, 325, 339, 392, 109, 15, 114, 737, 66, 108, 71, 59, 245, 195, 22])\n",
      "(1700000, [532, 413, 1128, 566, 13, 18, 183, 94, 102, 12, 235, 433, 16, 3, 32, 653, 18, 471, 532, 413, 110, 1128, 566, 10, 32, 532, 413, 268, 123, 130, 268, 93, 74, 515, 231, 1395, 110, 99, 19, 131, 145, 235, 433, 16, 11, 86, 253, 30, 34, 24, 57, 11, 30, 34, 50, 25, 31, 29, 28, 20, 40, 25, 11, 34, 14, 21, 14, 45, 52, 39, 91, 106, 46, 120, 50, 25, 36, 21, 21, 55, 185, 38, 38, 55, 120, 14, 359, 24, 46, 35, 359, 57, 106, 401, 106, 35, 42, 57, 90, 38, 55, 120, 14, 359, 24, 46, 35, 55, 36, 55, 38, 52, 30, 34, 38, 440, 390, 251, 380, 23, 390, 251, 318, 37, 478, 251, 79, 39, 38, 253, 35, 52, 39, 91, 25, 11, 55, 57, 52, 21, 24, 46, 50, 25, 36, 21, 21, 55, 185, 38, 38, 79, 31, 35, 359, 401, 30, 90, 79, 35, 42, 57, 90, 38, 23, 20, 23, 23, 40, 37, 20, 598, 37, 40, 29, 20, 28, 23, 40, 23, 31, 252, 228, 27, 31, 23, 23, 29, 252, 426, 457, 252, 28, 20, 40, 228, 27, 426, 40, 27, 40, 37, 45, 31, 455, 598, 31, 45, 426, 29, 457, 252, 45, 37, 29, 23, 31, 45, 28, 29, 23, 598, 44, 31, 44, 44, 455, 455, 228, 455, 25, 11, 34, 14, 21, 14, 45, 52, 57, 106, 46, 42, 24, 106, 46, 120, 50, 25, 36, 21, 21, 55, 185, 38, 38, 253, 35, 359, 57, 106, 401, 106, 35, 42, 57, 90, 38, 253, 384, 52, 36, 57, 39, 38, 30, 34, 384, 440, 390, 251, 380, 23, 390, 251, 318, 37, 478, 251, 79, 39, 35, 36, 21, 90, 120, 25, 11, 34, 14, 21, 14, 45, 83, 14, 90, 24, 50, 25, 143, 109, 156, 157, 11, 532, 110, 1922, 2236, 11, 860, 2545, 18, 480, 25, 85, 86, 38, 253, 30, 34, 24, 57, 85])\n",
      "(1800000, [428, 530, 56, 75, 1981, 461, 1044, 2051, 333, 56, 136, 16, 3])\n",
      "(1900000, [243, 107, 12, 392, 1362, 104, 87, 641, 59, 560, 158, 12, 392, 1362, 291, 188, 131, 43, 328, 525, 580, 220, 644, 16, 3, 19, 18, 43, 260, 150, 51, 392, 10, 87, 641, 59, 1212, 157, 115, 12, 10, 19, 18, 43, 508, 468, 1127, 12, 10, 190, 337, 505, 268, 110, 18, 43, 153, 732, 129, 392, 1362, 10, 291, 188, 200, 118, 10, 131, 43, 328, 525, 580, 19, 220, 526, 16])\n",
      "(2000000, [31, 23, 546, 10, 227, 10, 18, 180, 229, 191, 10, 671, 129, 153, 171, 894, 51, 95, 166, 65, 16, 3, 227, 10, 56, 75, 427, 129, 668, 62, 10, 18, 180, 229, 191, 10, 153, 115, 657, 854, 10, 71, 59, 1644, 53, 317, 10, 149, 32, 15, 97, 190, 178, 47, 535, 26, 22, 97, 153, 171, 51, 95, 166, 89, 13, 10, 1449, 229, 405, 1263, 10, 19, 59, 19, 41, 494, 171, 18, 118, 22])\n",
      "(2100000, [96, 209, 17, 434, 62, 1097, 536, 213, 404, 10, 115, 272, 62, 406, 693, 134, 98, 10, 151, 33, 1463, 1058, 12, 744, 189, 1097, 536, 104, 477, 459, 94, 33, 105, 16, 3, 96, 209, 434, 62, 1097, 536, 213, 404, 10, 115, 272, 62, 93, 693, 134, 10, 258, 121, 1463, 1058, 12, 744, 189, 477, 459, 104, 1097, 536, 94, 33, 245, 16])\n",
      "(2200000, [223, 283, 87, 65, 12, 765, 259, 230, 160, 87, 413, 191, 58, 186, 1736, 136, 16, 3, 19, 18, 765, 259, 454, 160, 10, 89, 74, 725, 291, 1062, 1136, 10, 89, 74, 725, 291, 893, 116, 152, 12, 222, 10, 125, 146, 61, 153, 186, 1736, 163, 140, 533, 18, 171, 10, 58, 894])\n",
      "(2300000, [357, 359, 46, 30, 83, 203, 38, 495, 4254, 1994, 1599, 94, 33, 102, 16, 3])\n",
      "(2400000, [856, 410, 104, 930, 930, 153, 298, 26, 10, 54, 295, 265, 136, 16, 3, 32, 294, 80, 104, 930, 930, 12, 223, 283, 63, 87, 65, 10, 105, 266, 221, 41, 1050, 73, 13, 186, 1736, 10, 125, 13, 153, 856, 856, 53, 66, 10, 103, 141, 13, 15, 13, 15, 295, 265, 347, 438, 45, 35, 45, 435])\n",
      "(2500000, [1453, 758, 12, 614, 345, 75, 104, 174, 729, 75, 497, 12, 174, 729, 330, 284, 129, 63, 129, 186, 498, 96, 138, 16, 3, 155, 41, 155, 354, 453, 12, 13, 174, 729, 75, 284, 129, 10, 75, 124, 511, 282, 15, 583, 10, 450, 608, 95, 97, 572, 238, 1453, 758, 12, 614, 345, 75, 10, 125, 13, 614, 345, 75, 13, 32, 484, 157, 178, 361, 75, 497, 10, 15, 127, 148, 369, 89, 288, 94, 33, 102, 187, 219, 163, 140, 210, 75, 99, 15, 97, 406, 453, 320, 157, 255, 174, 729, 75, 12, 284, 129, 26, 217])\n",
      "(2600000, [147, 234, 383, 651, 56, 1117, 372, 10, 1071, 1630, 28, 575, 124, 134, 142, 33, 16, 3, 383, 651, 919, 372, 10, 1071, 1630, 28, 100, 32, 168, 78, 28, 23, 23, 23, 840, 10, 1413, 184, 19, 183, 78, 4097, 142, 12, 1100, 170, 11, 1071, 1630, 28, 100, 32, 575, 124, 352, 142, 33, 16])\n",
      "(2700000, [538, 109, 548, 365, 179, 68, 3, 69, 411, 411, 411, 22, 31, 20, 28, 27, 27, 27, 22, 228, 402, 390, 70])\n",
      "(2800000, [137, 238, 32, 798, 10, 241, 192, 94, 102, 615, 636, 150, 577, 84, 137, 584, 740, 12, 75, 340, 16, 3, 279, 279, 508, 322, 322, 429, 118, 542, 177, 12, 870, 22, 22, 22, 155, 41, 74, 159, 263, 15, 542, 54, 33, 569, 10, 303, 1270, 186, 498, 214, 348, 11, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 17, 140, 124, 32, 54, 43, 604, 1226, 545, 47, 10, 261, 616, 114, 1716, 933, 264, 99, 60, 153, 53, 10, 125, 13, 17, 140, 124, 100, 32, 12, 150, 577, 78, 615, 636, 65, 10, 15, 184, 73, 1033, 26, 604, 1846, 10, 323, 352, 1039, 177, 12, 1035, 259, 1905, 761, 22, 22, 22, 11, 105, 19, 17, 12, 338, 607, 13, 557, 690, 22, 22, 22, 708, 184, 338, 159, 163, 140, 286, 740, 15, 56, 125, 17, 631, 116, 15, 73, 429, 1007, 54, 43, 856, 97, 22, 11, 168, 13, 15, 97, 202, 58, 15, 65, 12, 150, 577, 261, 103, 141, 93, 1424])\n",
      "(2900000, [15, 13, 291, 223, 284, 129, 10, 97, 283, 571, 75, 340, 341, 337, 204, 152, 175, 12, 127, 504, 10, 19, 67, 33, 476, 674, 68, 3, 204, 1445, 284, 129, 10, 125, 116, 341, 337, 204, 87, 163, 646, 813, 10, 1002, 299, 776, 821, 792, 513, 15, 13, 121, 177, 10, 476, 115, 411, 57, 46, 34, 55, 46, 24, 52, 52, 790, 663, 677, 524, 10, 107, 242, 179, 392, 10, 153, 115, 51, 12, 42, 90, 55, 37, 200, 330, 12, 51, 651, 299, 10, 274, 267, 1441, 2810, 283, 571, 10, 132, 578, 15, 513, 986, 1142, 651, 299, 10, 1190, 242, 55, 21, 193, 73, 171, 585, 73, 178, 361, 75, 22, 163, 140, 99, 13, 810, 980, 12, 325, 339, 10, 94, 102, 261, 103, 141, 328, 234, 18, 644, 22, 22])\n",
      "(100000, ';label_list_dense:', [1603, 1237, 826])\n",
      "(200000, ';label_list_dense:', [946, 19, 180])\n",
      "(300000, ';label_list_dense:', [734, 217, 36])\n",
      "(400000, ';label_list_dense:', [1988, 242])\n",
      "(500000, ';label_list_dense:', [1067])\n",
      "(600000, ';label_list_dense:', [659, 1215, 491, 15])\n",
      "(700000, ';label_list_dense:', [756, 1422, 380, 196, 848])\n",
      "(800000, ';label_list_dense:', [414, 201, 30, 4, 163])\n",
      "(900000, ';label_list_dense:', [873])\n",
      "(1000000, ';label_list_dense:', [271, 448, 40, 557])\n",
      "(1100000, ';label_list_dense:', [1176])\n",
      "(1200000, ';label_list_dense:', [1241])\n",
      "(1300000, ';label_list_dense:', [567, 4, 12, 2])\n",
      "(1400000, ';label_list_dense:', [526, 934, 12])\n",
      "(1500000, ';label_list_dense:', [1163])\n",
      "(1600000, ';label_list_dense:', [320, 71, 0, 7, 70])\n",
      "(1700000, ';label_list_dense:', [503, 387, 269])\n",
      "(1800000, ';label_list_dense:', [1475])\n",
      "(1900000, ';label_list_dense:', [872, 317, 71, 241, 70])\n",
      "(2000000, ';label_list_dense:', [637, 1825])\n",
      "(2100000, ';label_list_dense:', [527])\n",
      "(2200000, ';label_list_dense:', [696, 30, 383])\n",
      "(2300000, ';label_list_dense:', [457, 1890])\n",
      "(2400000, ';label_list_dense:', [1385, 2, 3, 35])\n",
      "(2500000, ';label_list_dense:', [79, 54])\n",
      "(2600000, ';label_list_dense:', [794, 1560, 668])\n",
      "(2700000, ';label_list_dense:', [1716])\n",
      "(2800000, ';label_list_dense:', [1585, 50])\n",
      "(2900000, ';label_list_dense:', [504, 28, 249, 71, 128])\n",
      "('num_examples:', 2999966, ';X.shape:', (2999966, 200), ';Y.shape:', (2999966, 1999))\n",
      "('train_X:', (2959966, 200), ';train_Y:', (2959966, 1999), ';vaild_X.shape:', (20000, 200), ';valid_Y:', array([[0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       ...,\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 1., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.]]), ';test_X:', (20000, 200), ';test_Y:', (20000, 1999))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "save cache files to file system successfully!\n"
     ]
    }
   ],
   "source": [
    "# generate training/validation/test data using source file and vocabulary/label set.\n",
    "#  get X,Y---> shuffle and split data----> save to file system.\n",
    "test_mode=False\n",
    "label_size=len(label2index)\n",
    "cache_path_h5py=base_path+'data.h5'\n",
    "cache_path_pickle=base_path+'vocab_label.pik'\n",
    "max_sentence_length=200\n",
    "\n",
    "# step 1: get (X,y) \n",
    "X,Y=get_X_Y(train_data_x,train_data_y,label_size,test_mode=test_mode)\n",
    "\n",
    "# pad and truncate to a max_sequence_length\n",
    "X = pad_sequences(X, maxlen=max_sentence_length, value=0.)  # padding to max length\n",
    "\n",
    "# step 2. shuffle, split,\n",
    "xy=list(zip(X,Y))\n",
    "random.Random(10000).shuffle(xy)\n",
    "X,Y=zip(*xy)\n",
    "X=np.array(X); Y=np.array(Y)\n",
    "num_examples=len(X)\n",
    "num_valid=20000\n",
    "num_valid=20000\n",
    "num_train=num_examples-(num_valid+num_valid)\n",
    "train_X, train_Y=X[0:num_train], Y[0:num_train]\n",
    "vaild_X, valid_Y=X[num_train:num_train+num_valid], Y[num_train:num_train+num_valid]\n",
    "test_X, test_Y=X[num_train+num_valid:], Y[num_train+num_valid:]\n",
    "print(\"num_examples:\",num_examples,\";X.shape:\",X.shape,\";Y.shape:\",Y.shape)\n",
    "print(\"train_X:\",train_X.shape,\";train_Y:\",train_Y.shape,\";vaild_X.shape:\",vaild_X.shape,\";valid_Y:\",valid_Y.shape,\";test_X:\",test_X.shape,\";test_Y:\",test_Y.shape)\n",
    "\n",
    "# step 3: save to file system\n",
    "save_data(cache_path_h5py,cache_path_pickle,word2index,label2index,train_X,train_Y,vaild_X,valid_Y,test_X,test_Y)\n",
    "print(\"save cache files to file system successfully!\")\n",
    "\n",
    "del X,Y,train_X, train_Y,vaild_X, valid_Y,test_X, test_Y\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TODO 1: use topic information\n",
    "below are some of things you can do, to have a better model.\n",
    "\n",
    "if you want to get better performance, you can use pre-trained word embedding and char embedding. \n",
    "\n",
    "addtionally,  if you want to model this task in a better way, you can use topic information. you can find it in topic_info.txt,  \n",
    "\n",
    "where each topic is assocate:\n",
    "\n",
    "this its parent topics(zeor,one or more); \n",
    "\n",
    "charactor tokens of topic's name; \n",
    "\n",
    "word tokens of topic's name;\n",
    "\n",
    "charactor tokens of topic's description; \n",
    "\n",
    "word tokens of topic's description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>738845194850773558</th>\n",
       "      <th>-5833678375673307423</th>\n",
       "      <th>c0,c1</th>\n",
       "      <th>w0</th>\n",
       "      <th>c0,c1,c2,c3,c4,c5,c6,c7,c0,c1,c8,c9,c10,c11,c12,c13,c14,c15,c16,c11,c17,c18,c19,c20,c21,c22,c23,c24,c25,c26,c27,c28,c29,c30,c20,c31,c24,c25,c26,c27,c11,c24,c32,c33,c34,c35,c36,c31,c8,c37,c38</th>\n",
       "      <th>w0,w1,w2,w3,w4,w5,w6,w7,w8,w9,w10,w11,w12,w13,w14,w15,w16,w17,w18,w15,w6,w19,w20,w21,w22,w23</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3738968195649774859</td>\n",
       "      <td>2027693463582123305</td>\n",
       "      <td>c39,c40</td>\n",
       "      <td>w24</td>\n",
       "      <td>c41,c42,c43,c39,c40,c4,c44,c45,c46,c47,c48,c49...</td>\n",
       "      <td>w24,w25,w26,w27,w28,w6,w29,w30,w11,w31,w32,w33...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4738849194894773882</td>\n",
       "      <td>1127459907694805235</td>\n",
       "      <td>c172,c31,c0,c1</td>\n",
       "      <td>w102</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7739004195693774975</td>\n",
       "      <td>2904932941037075699,1160326435131345730,725917...</td>\n",
       "      <td>c39,c40,c5,c173</td>\n",
       "      <td>w103</td>\n",
       "      <td>c39,c40,c23,c21,c174,c74,c5,c173,c17,c35,c39,c...</td>\n",
       "      <td>w104,w105,w11,w21,w24,w6,w106,w23,w54,w24,w107...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-7261194805221226386</td>\n",
       "      <td>-5833678375673307423</td>\n",
       "      <td>c36,c31,c45,c237</td>\n",
       "      <td>w148</td>\n",
       "      <td>c238,c239</td>\n",
       "      <td>w149,w150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-3689337711138901728</td>\n",
       "      <td>-2689200710357900655,-1689319711084901730</td>\n",
       "      <td>c215,c147,c105,c284,c97,c97,c168,c101,c146,c14...</td>\n",
       "      <td>w205,w54,w206</td>\n",
       "      <td>c196,c197,c0,c1,c313,c314,c315,c316,c317,c200,...</td>\n",
       "      <td>w125,w207,w208,w209,w166,w167,w23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    738845194850773558                               -5833678375673307423  \\\n",
       "0  3738968195649774859                                2027693463582123305   \n",
       "1  4738849194894773882                                1127459907694805235   \n",
       "2  7739004195693774975  2904932941037075699,1160326435131345730,725917...   \n",
       "3 -7261194805221226386                               -5833678375673307423   \n",
       "4 -3689337711138901728          -2689200710357900655,-1689319711084901730   \n",
       "\n",
       "                                               c0,c1             w0  \\\n",
       "0                                            c39,c40            w24   \n",
       "1                                     c172,c31,c0,c1           w102   \n",
       "2                                    c39,c40,c5,c173           w103   \n",
       "3                                   c36,c31,c45,c237           w148   \n",
       "4  c215,c147,c105,c284,c97,c97,c168,c101,c146,c14...  w205,w54,w206   \n",
       "\n",
       "  c0,c1,c2,c3,c4,c5,c6,c7,c0,c1,c8,c9,c10,c11,c12,c13,c14,c15,c16,c11,c17,c18,c19,c20,c21,c22,c23,c24,c25,c26,c27,c28,c29,c30,c20,c31,c24,c25,c26,c27,c11,c24,c32,c33,c34,c35,c36,c31,c8,c37,c38  \\\n",
       "0  c41,c42,c43,c39,c40,c4,c44,c45,c46,c47,c48,c49...                                                                                                                                               \n",
       "1                                                NaN                                                                                                                                               \n",
       "2  c39,c40,c23,c21,c174,c74,c5,c173,c17,c35,c39,c...                                                                                                                                               \n",
       "3                                          c238,c239                                                                                                                                               \n",
       "4  c196,c197,c0,c1,c313,c314,c315,c316,c317,c200,...                                                                                                                                               \n",
       "\n",
       "  w0,w1,w2,w3,w4,w5,w6,w7,w8,w9,w10,w11,w12,w13,w14,w15,w16,w17,w18,w15,w6,w19,w20,w21,w22,w23  \n",
       "0  w24,w25,w26,w27,w28,w6,w29,w30,w11,w31,w32,w33...                                            \n",
       "1                                                NaN                                            \n",
       "2  w104,w105,w11,w21,w24,w6,w106,w23,w54,w24,w107...                                            \n",
       "3                                          w149,w150                                            \n",
       "4                  w125,w207,w208,w209,w166,w167,w23                                            "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_info_data=pd.read_csv(base_path+'topic_info.txt', sep='\\t',encoding=\"utf-8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TODO 2: use both character and word tokens\n",
    "in this notebook we just use character tokens. it is fine. however, as many people observed, use word tokens to represent inputs, performance may be \n",
    "better. and if you can use both word and character tokens to represent inputs, performance can be much better. \n",
    "\n",
    "one of draw back to use word  is there are much more words then character. for example, in this task, total word token that frequency more than 5 is around 410k, while character token with frequency more than 5 is only around 11k. so much more memory is need. \n",
    "\n",
    "but you can still have a try if you want. with word token only, sequence length is shorter than character token, only about 50% length is needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TODO 3: use pre-trained character and word embedding\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
